We currently ingest records once the buffer reaches the threshold. The problem is we don't account for the incoming payload when deciding when to ingest.

Imagine we have ingested 7mb of data. The threshold is 8mb.

When a 2mb payload comes in, we append it and ingest. This means the payload was 9mb.

With these changes, we check if the incoming payload will put the buffer over the threshold before appending. If it does, we ingest the current buffer and then use the payload as the start of the new buffer.

This will likely cause more ingesting to occur, but it should mean ingest sizes are smaller and we don't hit issues with larger than expected payloads.